Stochastic Methods for Fine-Grained Image Segmentation and Uncertainty Estimation in Computer Vision

In this dissertation, we exploit concepts of probability theory, stochastic methods and machine learning to address three existing limitations of deep learning-based models for image understanding. First, although convolutional neural networks (CNN) have substantially improved the state of the art in image understanding, conventional CNNs provide segmentation masks that poorly adhere to object boundaries, a critical limitation for many potential applications. Second, training deep learning models requires large amounts of carefully selected and annotated data, but large-scale annotation of image segmentation datasets is often prohibitively expensive. And third, conventional deep learning models also lack the capability of uncertainty estimation, which compromises both decision making and model interpretability. To address these limitations, we introduce the Region Growing Refinement (RGR) algorithm, an unsupervised post-processing algorithm that exploits Monte Carlo sampling and pixel similarities to propagate high-confidence labels into regions of low-confidence classification. The probabilistic Region Growing Refinement (pRGR) provides RGR with a rigorous mathematical foundation that exploits concepts of Bayesian estimation and variance reduction techniques. Experiments demonstrate both the effectiveness of (p)RGR for the refinement of segmentation predictions, as well as its suitability for uncertainty estimation, since its variance estimates obtained in the Monte Carlo iterations are highly correlated with segmentation accuracy. We also introduce FreeLabel, an intuitive open-source web interface that exploits RGR to allow users to obtain high-quality segmentation masks with just a few freehand scribbles, in a matter of seconds. Designed to benefit the computer vision community, FreeLabel can be used for both crowdsourced or private annotation and has a modular structure that can be easily adapted for any image dataset. The practical relevance of methods developed in this dissertation are illustrated through applications on agricultural and healthcare-related domains. We have combined RGR and modern CNNs for fine segmentation of fruit flowers, motivated by the importance of automated bloom intensity estimation for optimization of fruit orchard management and, possibly, automatizing procedures such as flower thinning and pollination. We also exploited an early version of FreeLabel to annotate novel datasets for segmentation of fruit flowers, which are currently publicly available. Finally, this dissertation also describes works on fine segmentation and gaze estimation for images collected from assisted living environments, with the ultimate goal of assisting geriatricians in evaluating health status of patients in such facilities

Microscopia in situ para análise de Bactérias filamentosas: ótica e processamento de imagens

In the activated sludge process, problems of foaming and filamentous bulking can occur due to overgrowth of certain filamentous bacteria. Nowadays, these microorganisms are typically monitored by means of light microscopy combined with staining techniques. As drawbacks, these methods are susceptible to human errors, subjectivity and limited by the use of discontinuous microscopy. The present project aims the application of an in situ microscope (ISM) for continuous monitoring of filamentous bacteria, providing real-time examination, automated analysis and elimination of sampling, preparation and transport of samples. The ISM previously developed at the Hochschule Mannheim required adaptations for use within wastewater environment, specially in terms of impermeability and development of a cleaning mechanism. With a new objective lens design, the system was simplified to a single tubus and an externally activated cleaning system based on magnetism was created. A proper image processing algorithm was designed for automated recognition and measurement of filamentous objects, allowing real-time evaluation of images without any staining, phase-contrast or dilution techniques. Three main operations are performed: preprocessing and binarization; recognition of filaments using distance-maps and shape descriptors; measurement and display of total extended filament length. A 3D-printed prototype was used for experiments with respect to the new ISM’s design, providing images with resolution very close to the ones acquired with the previous microscope. The designed cleaning system has shown to be effective, removing dirt settled above the lens during tests. For evaluation of the image processing algorithm, samples from an industrial activated sludge plant were collected weekly for a period of twelve months and imaged without any prior conditioning, replicating real environment conditions. Experiments have shown that the developed algorithm correctly identifies trends of filament growth rate, which is the most important parameter for decision making. For reference images whose filaments were marked by specialists, the algorithm correctly recognized 72% of the filaments pixels, with a false positive rate of at most 14%. An average execution time of 0.7 second per image was achieved, demonstrating the algorithm suitability for real-time monitoring.CAPES; CNPqEm processos de lodo ativado, problemas de foaming e filamentous bulking podem ocorrer devido ao crescimento exagerado de bactérias filamentosas. Atualmente, o monitoramento de tais micro-organismos é feito por meio de métodos baseados em microscopia ótica combinada com técnicas de marcadores, os quais apresentam limitações intrínsecas da microscopia descontínua, são subjetivos e suscetíveis a erro humano. O presente projeto visa a aplicação de um microscópio in situ (ISM) para monitoramento contínuo de bactérias filamentosas, de forma a possibilitar análise instantânea, computadorizada, sem necessidades de recolher, preparar e transportar amostras. O ISM previamente desenvolvido na Hochschule Mannheim teve que ser adaptado para análise de águas residuais, especialmente em termos de impermeabilidade e a criação de um mecanismo de limpeza. Com a utilização de uma nova objetiva, o novo ISM foi simplificado para um tubo único e um sistema de limpeza ativado externamente baseado em magnetismo foi criado. Um algoritmo de processamento de imagens foi elaborado para reconhecimento e medição de comprimento de estruturas filamentosas, permitindo avaliação em tempo real de imagens sem qualquer técnica de marcadores, contraste de fase ou diluição. O mesmo consiste em três operações principais: pré-processamento e binarização; reconhecimento de filamentos por meio de mapeamento de dis- tâncias e descritores de forma; e, finalmente, medição e visualização do comprimento de cada filamento. Um protótipo construído via impressão 3D foi utilizado para avaliação o novo design do microscópio, fornecendo imagens com resolução bastante próxima das adquiridas com a versão anterior do sistema. O mecanismo de limpeza desenvolvido mostrou-se efetivo, capaz de remover partículas sedimentadas acima das lentes durante os testes. Para avaliação do algoritmo de processamento de imagens, amostras de uma planta industrial de lodo ativado foram coletadas semanalmente por um período de doze meses e imageadas sem qualquer condicionamento prévio, replicando condições reais de ambiente. Experimentos demonstraram que o algoritmo desenvolvido identifica corretamente tendências de aumento/decréscimo da concentração de filamentos, o que constitui o principal parâmetro para tomadas de decisão. Para imagens de referência cujos filamentos foram marcados por especialistas, o algoritmo reconheceu corretamente 80% dos pixels atribuídos a filamentos, com uma taxa de falso positivos de até 24%. Um tempo de execução médio de 0,7 segundo por imagem foi obtido, provando sua aptidão para formar uma ferramenta de monitoramento em tempo real